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With the anticipated increase of small unmanned aircraft systems (sUAS) entering into 
the National Airspace System, it is highly likely that vehicle operators will be teaming with 
fleets of small autonomous vehicles. The small vehicles may consist of sUAS, which are 
55 pounds or less that typically will fly at altitudes 400 feet and below, and small ground 
vehicles typically operating in buildings or defined small campuses. Typically, the vehicle 
operators are not concerned with manual control of the vehicle; instead they are concerned 
with the overall mission. In order for this vision of high-level mission operators working 
with fleets of vehicles to come to fruition, many human factors related challenges must 
be investigated and solved. First, the interface between the human operator and the au- 
tonomous agent must be at a level that the operator needs and the agents can understand. 
This paper details the natural language human factors efforts that NASA Langley’s Au- 
tonomy Incubator is focusing on. In particular these efforts focus on allowing the operator 
to interact with the system using speech and gestures rather than a mouse and keyboard. 
With this ability of the system to understand both speech and gestures, operators not 
familiar with the vehicle dynamics will be able to easily plan, initiate, and change missions 
using a language familiar to them rather than having to learn and converse in the vehicle’s 
language. This will foster better teaming between the operator and the autonomous agent 
which will help lower workload, increase situation awareness, and improve performance of 
the system as a whole. 


Nomenclature 


UsV Unmanned Vehicle 

GCS Ground Control Station 

NL Natural Language 

SAR Search and Rescue 

sUAS Small Unmanned Aerial System 

sUAV Small Unmanned Aerial Vehicle 


I. Introduction 

S MALL unmanned aerial vehicles (sUAV) are starting to become ubiquitous because they are relatively 
cheap and are fairly easy to fly while the potential immediate productivity gain is large for applications 
such as photography and inspection. As more people find innovative ways to employ sUAVs - such as crop 
monitoring, 2 photography ' and filming, package delivery, 11 pipeline inspection, * 1 ’ search and rescue (SAR), 
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and fire monitoring just to name a few - the way humans interact with them will become critical. Current 
interaction methods typically include manual controllers, ’ 1 smartphones and tablets, or graphical ground 
control stations (GCS). For the graphical GCSs that allow waypoint navigation, 1 ’ several freeware versions 
are available (e.g., Mission Planner, APM Planner 2, MAVProxy, Tower (DroidPlanner 3), AndroPilot, 
MAVPilot, iDroneCtrl, and QGroundControl). Interacting, though, with all these types of controllers 
requires the operator to learn and understand the sUAV dynamic behavior rather than having a more natural 
and teaming relationship. A lack of teaming typically results in increased workload, decreased situation 
awareness, and trust issues among all active agents. However, with the possibility of communicating with 
various types of unmanned vehicles (Uj:Vs) by more natural language (NL) methods, such as speech and 
gestures, the teaming aspect may come to full fruition. 

The capabilities of not only small unmanned aerial systems (sUAS) but also the systems for all UzVs, be it 
a more sophisticated sensor suite, awareness of internal state and surrounding environment, decision making 
under increasing uncertainty, are rapidly advancing. In order to make the most effective and efficient use of 
these advances, true human-machine teaming is necessary. If architectures and requirements are developed 
from the start with teaming as a goal, increasing autonomous vehicle system capabilities, including sUAS 
capabilities, would naturally integrate with the human operator. 

This paper will detail the considerations of operators, who may not fully understand the dynamics of 
the sUAVs under their control, interacting with autonomous agents using NL in order to increase teaming 
between them and the sUAVs. Possible solutions with then be discussed. Lastly, this research is part of 
a corpus of work NASA Langley Research Center’s Autonomy Incubator which includes an architectural 
backbone 1 that is able to seamlessly encompass machine learning ’ and decision making besides this 
natural language work while ensuring safe flight within an organic environment. - ’ 

II. Problem Setting 

For UiVs and possibly for personnel air vehicles, the operator may be partially or completely unaware 
of the dynamics of the vehicle and its environment. For example, a user controlling a sUAV may not fully 
understand traditional stick and rudder concepts nor have a basic understanding of aerodynamics such as 
how stalls occur. Furthermore, the operator may understand that certain expanses need to be searched for a 
SAR mission; not how to develop the optimal routing for the vehicle or how to maintain the flight path and 
sUAV attitude. For operational safety, the vehicles and operators should have certain characteristics and 
capabilities 12 and the interface between the two should also have certain characteristics and capabilities. 

A. UccV Operators 

As mentioned above, operators may not fully understand the dynamics of the vehicles they are managing. 
Instead, they may be experts in the overall mission, such as how best to conduct a SAR or inspection mission, 
or dispatching Ua;Vs for package delivery. As UxVs become more autonomous, the relationship between the 
operators and the vehicles will change from them behaving as managers to becoming more team members 
with the autonomous vehicles. 

1. Current Interaction with sUAVs 

The current interactions of operators with sUAVs typi- 
cally have them in the role of the pilot where they have di- 
rect input to the attitude of the sUAV. Their interactions, 
though, have steadily moved from single- to multi-loop 
control (comprised of inner- and outer- loop controllers). 

For basic inner-loop control of the sUAV, operators must 
have a better understanding of the dynamics of their ve- 
hicles and they fly using more traditional stick and rudder 
controllers (Fig. 1). In fact, the GCS may be comprised 
solely of this type of controller. 

However, with operators becoming more of mission 
managers or team members with other autonomous agents, they no longer are concerned about actively 
maintaining the attitude of the sUAV. Instead, their primary concern isare what the vehicle is doing in 
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Figure 1. Typical sUAV manual controller. 
The sUAV operator directly controls the rudder, 
throttle, elevator, and aileron. 
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relationship to the overall mission. As such, most sUAVs now have autonomatic stabilization and several 
are able to do waypoint navigation ’ using graphical user interfaces 1 1 (Fig. 2). Even with the advent of 
more sophisticated autopilots, the operator is still tasked with maintaining basic flight of the sUAV through 
waypoint setting and constant monitoring. 
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Figure 2. Typical graphical GCS to plan and control a sUAV mission via waypoints. Figure is the Mission Plan- 
ner interface from http:/ /planner. ardupilot.com/wp-content/uploads/sites/5/2014/04/MP-FP-Screen.jpg. 


2. Teaming 

As UzVs become more autonomous, the UxV operator’s role will become that of a mission manager and an 
active team member with the autonomous UxV. As mission managers, the operators must understand the 
goals and subgoals of the mission and how to accomplish those within the confines of their knowledge and the 
interface ( i.e ., the GCS) between them and the vehicle. These mission goals must be deary communicated 
to the vehicle by the operator. Mutual understanding of the mission goals is a requirement for the operators 
to become participating team members with the autonomous agents under their control. 

Other requirements to be a successful team member besides understanding the mission goals include 
communication and trust among all team members. To facilitate trust, a shared understanding and situation 
awareness among all parties is a must. With the goal of autonomy, these characteristics must now also be 
inherent in the autonomous Ua;Vs for optimal performance of the team. 

B. Missions 

Three classes of missions that incorporate several capabilities the system (humans + GCS + autonomous 
agents) must possess to perform optimally are SAR and inspection missions, data sampling operations, and 
delivering packages. This section will describe requirements for the operator interface to the autonomous 
agents in order to define and manage these classes of missions. 

1. Search and Rescue and Inspection 

SAR and inspection missions are very similar. Inspection missions include inspecting power lines, pipelines, 1 
oil rigs, buildings, etc. Both these types of mission have a defined volume of interest, particular patterns the 
UxV needs to follow, and objects of interest. The volume of interest may be very confined, such as inside 
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a transformer station, or very large, such as a mountain range, but there is a definable area of interest. 
sUAVs are well suited for many of these missions and the patterns the sUAV needs to fly are typically well 
defined. These patterns consist of following a trail, road, or power- or pipe-line, and grid or circular patterns 
typically used in SAR. ’ Lastly, the objects of interest are known and may be a lost hiker, a downed wire, 
etc. Therefore, to conduct these types of missions, the operator must define the search or inspection volume, 
the pattern to be followed, and ensure the sensors required to look for the signals of interest are available 
on the UxV. The operator’s input of the area of interest and the flight path typically involves manual entry 
into the GCS which usually requires the operator to define each waypoint along the path. 

Tactical changes to these types of missions may be cumbersome with the traditional manual controller or 
GCS. Tactical changes will usually arise from either changing the pattern between the grid and concentric 
circle patterns, or wanting to hover or circle around an object of interest. 

2. Data Sampling 

Data sampling consists of missions where the sUAV is 
collecting samples along its flight path. An example mis- 
sion would be coordinating several sUAVs to take CO 2 air 
samples continuously in a coordinated pattern of concen- 
tric circles while increasing altitude at a particular rate 
(Fig. 3). 26, As with SAR and inspection missions, these 
types of operations require defining a volume of interest 
and flight patterns. Some of the desired flight patterns 
would require the manual input of several waypoints to 
fully define the path. The operator would also have to in- 
put other parameters such as required UxV coordination 
and UxV vehicle speeds. 

Tactical changes to data sampling missions may be 
driven by the desire to follow an object of interest ( e. < 7 . , a 
smoke plume). For this type of change, easily restricting 
the sUAV operating bounds through the traditional GCS 
may be cumbersome. 

3. Package Delivery 

Defining a package delivery mission is substantially different from specifying the previous missions described. 
Typically, this type of mission requires starting, drop off, and return (if different from the starting) locations 
for the sUAV, a delivery time, and the possibility of exclusion areas, such as schools. Of these requirements, 
exclusion areas may be the most difficult to input using the traditional GCS. 

Package delivery mission tactical changes will often involve changes in the delivery location due to receiver 
preferences ( e.g ., the back door rather than the front stoop) or unavailability to the original drop-off location 
due to unexpected and mobile obstacles like pets or vehicles. Once again, either manually controlling the 
UirV (if it is a point-to-point delivery system that starts in a warehouse with the delivery point as an office) 
in real time to affect these changes, or trying to define new waypoints or drop off locations in real time may 
be cumbersome with the traditional GCS. 

III. Approach: Natural Language Interaction to Facilitate Teaming 

In addition to increasing the autonomy on the vehicle to plan new trajectories" ' ’ and avoid obstacles , !0 
improving the user interface will facilitate the communication of any changes to both the operator and the 
other autonomous agents. This improved communication amongst all agents will benefit transparency and 
predictability. This, is turn, will promote the sense of all agents as being part of a team. 

With communication among all autonomous entities facilitating teaming, a shared perception among all 
agents will enable the operator to more readily trust the autonomous UzVs ;' 51 to achieve this, the operator 
must understand what each vehicle is doing and why it is doing it ( i.e ., intent). Possible methods to show the 
operator the decision state of the autonomous vehicle includes directed graphs and augmented reality. " ~ 1 1 



Figure 3. Example sUAV coordinated helix flight 
pattern for an air data sampling mission. 
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More streamlined means of the operator communicating with the vehicles include speech and gestures. 
Using these methods, which are natural to the operator, will decrease workload, increase situation aware- 
ness, and increase communication to the autonomous agents. This allows the operator to focus on mission 
execution (rather than the minutiae of changes) which, by extension, will increase teaming and trust in the 
system. 

A. Speech Recognition 

Using speech recognition to give commands is becoming pervasive, especially as speech commands move 
from controlled to natural language. Many people are now comfortable with speech commands beyond the 
primitive phone tree systems, such as Apple’s SIRI, Microsoft’s speech recognition system, and Google’s 
Now. Great progress has been made in these systems to understand human speech without training. 
However, the word error rate is still rather high for typical conversational speech recognition especially 
in noisy environments. Various methods to improve on this have been implemented (for example, see 
McMillian and Gilbert (2008) ) but have only been partially successful. 

The use of speech recognition systems to control U:rVs would allow operators to more naturally com- 
municate missions to the system. Even under current NL instantiations, effective mission directions for 
UxVs are likely possible. The environments these operators would be working in would likely be relatively 
quiet or the operator may have the ability to filter out some external noise by going into an enclosed space 
(for example a car). Furthermore, the commands given to the system would typically fall within a certain 
vocabulary scheme - a controlled language. Incorporating speech commands rather than using the more 
traditional mouse and keyboard entry methods may increase the speed of response, which is critical for SAR 
missions, and subsequently decrease operator stress levels. 

For SAR and inspection missions, typical commands would encompass the pattern to follow such as a 
grid or concentric circles. The mission operator would also define starting and end points, especially for pipe- 
or power-line inspection, and the sUAV’s altitude. Lastly, the operator would define the overlap required to 
ensure full coverage. See Table 1 on the next page for suggested voice commands for SAR and inspection 
missions. Of course, for this mode to be more natural to the operator, the commands could be given in any 
order; for example a , 

Fly a GRID pattern at 400 FEET with 30 # /« OVERLAP. 

START at your CURRENT LOCATION and fly a PATTERN LENGTH of 100 YARDS before turning around. 

END and RETURN once you have flown out 400 YARDS. 

Similar verbal commands could be seen for other missions. For example, atmospheric data sampling 
mission commands could be (see Fig. 3 on the previous page for an example graphical representation) 

With three drones, fly CONCENTRIC HELIX patterns FROM THE GROUND up TO 400 FEET 
with the SAME HEADINGS. 

START at your CURRENT LOCATION and the FIRST DRONE should fly STRAIGHT UP. 

The SECOND UAV should fly a helix with a 5 FOOT RADIUS. 

And the LAST DRONE should use a RADIUS of 10 FEET. 

GO UP at 2 FEET PER SECOND. 

A package delivery verbal mission definition should encompass which package to deliver and the drop off 
location rather than the flight pattern. An example package delivery verbal mission definition could be 

Deliver PACKAGE 3U56TS from the DISTRIBUTION CENTER TO BUILDING 2101 NOW and RETURN. 

Work in NASA Langley’s Autonomy Incubator is incorporating speech commands to both control a 
sUAV and to initially define and strategically change its mission. Currently, the Autonomy Incubator can 
use speech commands to takeoff, return, and land a sUAS. Additional work in this area includes broadening 
the range of commands the sUAV understands and using speech to define the initial parameters for SAR and 
package delivery missions. Subsequent work in this are may include incorporating speech generation from 
UxVs to the human team members to further increase the NL capabilities. 

CAPITALIZED RED text indicates key words. See Table 1 on the following page for example key words. 
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Table 1. Possible SAR and Inspection Speech Commands 


Category 

Command 

Options 

Example 

Pattern 

Grid 

Concentric Circles 

Hover 

Circle 



Area to be covered 

Start /Begin 

Current location 

start at current location 



Latitude and longitude 

start at 42.3598° N, 71.0921° W 



Named waypoint 

begin at BARN 


Pattern length 

x feet or yards 

pattern length of 400 yards 


End/Finish 

Latitude an longitude 

end at 42.3598° N, 71.0921° W 



Named waypoint 

finish at BARN 



Distance 

end 400 yards out 


Return 

End point 

do not return 



Named waypoint 

return to BARN 



Starting point 

return 




return to starting point 

Altitude 

At 

x feet 

at 400 feet 

at 400 feet altitude 

Amount of overlap 

Overlap 

x% 

overlap 30% 



x feet 

overlap 5 feet 


B. Gesture Recognition 

People naturally communicate using gestures such as pointing or indicating the size of an object. Tablet and 
smartphone users are becoming more comfortable with interacting with computers using gestures because 
of the touchscreens used on most of these devices. Users can now quickly activate a device by swiping the 
screen, and increase or decrease the image size on the screen with pinching actions. Gestures are a natural 
way for humans to communicate and the vast majority of gestures contain useful information. 

More free form gestures produced out in space rather than directly touching a screen are starting to 
be recognized and interpreted. Examples are Microsoft’s Kinect, Leap Motion Controller, and Intel’s Re- 
alSense. These technologies are rapidly improving 1 ~ and being used in a variety of settings in addtion to 

. 45—47 

gaming. 

The types of gestures for mission management would be emblematic, iconic, and deictic gestures. As with 
speech recognition systems, gesture recognition systems could easily allow operators to define start and end 
locations (deictic gestures), and patterns to fly (both an emblematic and iconic gesture). Rather than having 
the operator tediously enter waypoints for an atmospheric data sampling mission, the operator could use 
gestures to indicate the pattern to be flown and if the underlying algorithms understood the environmental 
factors such as the location and volume boundaries, the system could translate the gestures to a flight path 
in the predefined volume. For example, the mission operator could gesture a helix pattern and the system 
could translate that into a flight path. For a SAR or inspection mission, the operator could gesture a grid 
pattern and the system could translate that into waypoints. 

While the optimal gesture recognition system would be external to the operators, thus leaving them 
unencumbered, some type of tactile feedback may be of benefit. For example, if a gesture in free space was 
to pan a map or flip a switch, some tactile feedback to indicate that the map or switch was grabbed rather 
than just visual feedback may enhance the NL aspects of the GCS. 

Current work in the Autonomy Incubator is incorporating gesture recognition with tactile feedback to 
interact with displays in the GCS. Furthermore, for more tactical changes in the vehicle path that may 
require manual marking of local object interest, previous work by the author indicated that controlling a 
vehicle using hand motions is feasible. ' This work will eventually be incorporated into the GCS to allow 
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for real-time tactical changes during SAR and package delivery missions where the operator may control the 
sUAV to more closely look at an object of interest or control the sUAV during the “last 50 feet” of package 
delivery respectively. 

C. Natural Interaction (Speech and Gestures) 

The most natural way for people to communicate is with both speech and gestures; 1 in fact, a person 
may even gesture while speaking even if the receiver can not see the gestures ( e.g ., talking with someone 
on the phone). This concept started to take root with Bolt’s seminal paper “Put-That-There.” 5 " With 
the improvements in speech and gesture recognition systems, many are looking to combine both into a 
multimodal system in order to improve efficiency and decrease workload. 

With the increasing use of NL to define and control missions, the degree of true teaming may increase 
because the operator may feel more in touch with the autonomous agents. Furthermore, if the autonomous 
agents communicate back to the operator their understanding of the command, similar to what pilots do with 
air traffic control commands, the level of trust among all agents should increase due to increase transparency 
of the system. 

With this in mind, an example multimodal command for the SAR mission described above (Section III A 
on page 5) might be b 

Fly THIS [GESTURES A GRID PATTERN] at 400 FEET with 30% OVERLAP. 

START here [POINTS TO A LOCATION] and fly this far out [GESTURES TO A POINT IN THE DISTANCE] 
before turning around. 

END here [POINTS TO A LOCATION] and RETURN once you have flown out 
to there [GESTURES AT A LOCATION] . 

IV. Conclusion 

Humans naturally communicate with one another using both speech and gestures. In order to fully realize 
the benefits of working with autonomous agents such as sUAVs, these autonomous agents should understand 
the operator’s multimodal input. With such multimodal communication will come better teaming among 
all agents and if aided with transparency, will decrease workload, increase situation awareness, and increase 
efficiency and trust throughout the system. 

NASA LaRC’s Autonomy Incubator is currently working on incorporating voice and gesture commands 
in the GCS in order to control various UxVs. This research is initially focused on strategic mission defini- 
tion and eventually real-time tactical changes to the mission. The first missions considered include SAR, 
atmospheric data sampling, and package delivery since these tasks encompass commands common to various 
other missions (such as crop and wildlife monitoring, and aerial photography). The Autonomy Incubator’s 
goal is to combine the NL mission definition with appropriate feedback to enable trust in the system as a 
whole, with work encompassing algorithms to plan coordinated flight paths and real-time obstacle avoidance. 
With this integration, a mission manager will feel more like a leader of a team of agents, rather than a lone 
individual, with all involved striving to reach the same goal in a multi-task mission. 
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